Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses

BACKGROUND: Reliable performance assessment is a necessary prerequisite for outcome-based assessment of surgical technical skill. Numerous observational instruments for technical skill assessment have been developed in recent years. However, methodological shortcomings of reported studies might negatively impinge on the interpretation of inter-rater reliability. OBJECTIVE: To synthesize the evidence about the inter-rater reliability of observational instruments for technical skill assessment for high-stakes decisions. DESIGN: A systematic review and meta-analysis were performed. We searched Scopus (including MEDLINE) and Pubmed, and key publications through December, 2016. This included original studies that evaluated reliability of instruments for the observational assessment of technical skills. Two reviewers independently extracted information on the primary outcome (the reliability statistic), secondary outcomes, and general information. We calculated pooled estimates using multilevel random effects meta-analyses where appropriate. RESULTS: A total of 247 documents met our inclusion criteria and provided 491 inter-rater reliability estimates. Inappropriate inter-rater reliability indices were reported for 40% of the checklists estimates, 50% of the rating scales estimates and 41% of the other types of assessment instruments estimates. Only 14 documents provided sufficient information to be included in the meta-analyses. The pooled Cohen's kappa was.78 (95% CI 0.69-0.89, p < 0.001) and pooled proportion agreement was 0.84 (95% CI 0.71-0.96, p < 0.001). A moderator analysis was performed to explore the influence of type of assessment instrument as a possible source of heterogeneity. CONCLUSIONS AND RELEVANCE: For high-stakes decisions, there was often insufficient information available on which to base conclusions. The use of suboptimal statistical methods and incomplete reporting of reliability estimates does not support the use of observational assessment instruments for technical skill for high-stakes decisions. Interpretations of inter-rater reliability should consider the reliability index and assessment instrument used. Reporting of inter-rater reliability needs to be improved by detailed descriptions of the assessment process

Lysosomotropism depends on glucose : a chloroquine resistance mechanism

There has been long-standing interest in targeting pro-survival autophagy as a combinational cancer therapeutic strategy. Clinical trials are in progress testing Chloroquine (CQ) or its derivatives in combination with chemo- or radio-therapy for solid and haematological cancers. While CQ has shown efficacy in pre-clinical models, its mechanism of action remains equivocal. Here, we tested how effectively CQ sensitises metastatic breast cancer cells to further stress conditions such as ionising irradiation, doxorubicin, PI3K-Akt inhibition and serum withdrawal. Contrary to the conventional model, the cytotoxic effects of CQ were found to be autophagy-independent, since genetic targeting of ATG7 or the ULK1/2 complex could not sensitise cells, like CQ, to serum depletion. Interestingly, while CQ combined with serum starvation was robustly cytotoxic, further glucose starvation under these conditions led to a full rescue of cell viability. Inhibition of hexokinase using 2-deoxyglucose (2DG) similarly led to CQ resistance. As this form of cell death did not resemble classical caspase-dependent apoptosis, we hypothesised that CQ-mediated cytotoxicity was primarily via a lysosome-dependent mechanism. Indeed, CQ treatment led to marked lysosomal swelling and recruitment of Galectin3 to sites of membrane damage. Strikingly, glucose starvation or 2DG prevented CQ from inducing lysosomal damage and subsequent cell death. Importantly, we found that the related compound, amodiaquine, was more potent than CQ for cell killing and not susceptible to interference from glucose starvation. Taken together, our data indicate that CQ effectively targets the lysosome to sensitise towards cell death but is prone to a glucose-dependent resistance mechanism, thus providing rationale for the related compound amodiaquine (currently used in humans) as a better therapeutic option for cancer

A computerised test of perceptual ability for learning endoscopic and laparoscopic surgery and other image guided procedures: Score norms for PicSOr

Background: The aptitude to infer the shape of 3-D structures, such as internal organs from 2-D monitor displays, in image guided endoscopic and laparoscopic procedures varies. We sought both to validate a computer-generated task Pictorial Surface Orientation (PicSOr), which assesses this aptitude, and to identify norm referenced scores. Methods: 400 subjects (339 surgeons and 61 controls) completed the PicSOr test. 50 subjects completed it again one year afterwards. Results: Complete data was available on 396 of 400 subjects (99%). PicSOr demonstrated high test and re-test reliability (r = 0.807, p < 0.000). Surgeons performed better than controls' (surgeons = 0.874 V controls = 0.747, p < 0.000). Some surgeons (n = 22–5.5%) performed atypically on the test. Conclusions: PicSOr has population distribution scores that are negatively skewed. PicSOr quantitatively characterises an aptitude strongly correlated to the learning and performance of image guided medical tasks. Most can do the PicSOr task almost perfectly, but a substantial minority do so atypically, and this is probably relevant to learning and performing endoscopic tasks

Validation studies of virtual reality simulation performance metrics for mechanical thrombectomy in ischemic stroke

Introduction: Mechanical thrombectomy (MT) has transformed the treatment of ischemic stroke. However, patient access to MT may be limited due to a shortage of doctors specifically trained to perform MT. The studies reported here were done to (1) develop, operationally define, and seek consensus from procedure experts on the metrics which best characterize a reference procedure for the performance of an MT for ischemic stroke and (2) evaluate their construct validity when implemented in a virtual reality (VR) simulation. Methods: In study 1, the metrics for a reference approach to an MT procedure for ischemic stroke of 10 phases, 46 steps, and 56 errors and critical errors, were presented to an international Delphi panel of 21 consultant level interventional neuroradiologists (INRs). In study 2, the metrics were used to assess 8 expert and 10 novice INRs performing a VR simulated routine MT procedure. Results: In study 1, the Delphi panel reached consensus on the appropriateness of the procedure metrics for a reference approach to MT in ischemic stroke. Group differences in median scores in study 2 demonstrated that experienced INRs performed the case 19% faster (P=0.029), completed 40% more procedure phases (P=0.009), 20% more steps (P=0.012), and made 42% fewer errors (P=0.016) than the novice group. Conclusions: The international Delphi panel agreed metrics implemented in a VR simulation of MT distinguished between the computer scored procedure performance of INR experts and novices. The studies reported here support the demonstration of face, content, and construct validity of the MT metrics

Acute surgical wound-dressing procedure: Description of the steps involved in the development and validation of an observational metric

The aim of this study was to develop an observational metric that could be used to assess the performance of a practitioner in completing an acute surgical wound-dressing procedure using aseptic non-touch technique (ANTT). A team of clinicians, academics, and researchers came together to develop an observational metric using an iterative six-stage process, culminating in a Delphi panel meeting. A scoping review of the literature provided a background empirical perspective relating to wound-dressing procedure performance. Video recordings of acute surgical wound-dressing procedures performed by nurses in clinical (n = 11) and simulated (n = 3) settings were viewed repeatedly and were iteratively deconstructed by the metric development group. This facilitated the identification of the discrete component steps, potential errors, and sentinel (serious) errors, which characterise a wound dressing procedure and formed part of the observational metric. The ANTT wound-dressing observational metric was stress tested for clarity, the ability to be scored, and interrater reliability, calculated during a further phase of video analysis. The metric was then subjected to a process of cyclical evaluation by a Delphi panel (n = 21) to obtain face and content validity of the metric. The Delphi panel deliberation verified the face and content validity of the metric. The final metric has three phases, 31 individual steps, 18 errors, and 27 sentinel errors. The metric is a tool that identifies the standard to be attained in the performance of acute surgical wound dressings. It can be used as both an adjunct to an educational programme and as a tool to assess a practitioner's performance of a wound-dressing procedure in both simulated and clinical practice contexts

The development and validation of the Virtual Tissue Matrix, a software application that facilitates the review of tissue microarrays on line

BACKGROUND: The Tissue Microarray (TMA) facilitates high-throughput analysis of hundreds of tissue specimens simultaneously. However, bottlenecks in the storage and manipulation of the data generated from TMA reviews have become apparent. A number of software applications have been developed to assist in image and data management; however no solution currently facilitates the easy online review, scoring and subsequent storage of images and data associated with TMA experimentation. RESULTS: This paper describes the design, development and validation of the Virtual Tissue Matrix (VTM). Through an intuitive HTML driven user interface, the VTM provides digital/virtual slide based images of each TMA core and a means to record observations on each TMA spot. Data generated from a TMA review is stored in an associated relational database, which facilitates the use of flexible scoring forms. The system allows multiple users to record their interpretation of each TMA spot for any parameters assessed. Images generated for the VTM were captured using a standard background lighting intensity and corrective algorithms were applied to each image to eliminate any background lighting hue inconsistencies or vignetting. Validation of the VTM involved examination of inter-and intra-observer variability between microscope and digital TMA reviews. Six bladder TMAs were immunohistochemically stained for E-Cadherin, β-Catenin and PhosphoMet and were assessed by two reviewers for the amount of core and tumour present, the amount and intensity of membrane, cytoplasmic and nuclear staining. CONCLUSION: Results show that digital VTM images are representative of the original tissue viewed with a microscope. There were equivalent levels of inter-and intra-observer agreement for five out of the eight parameters assessed. Results also suggest that digital reviews may correct potential problems experienced when reviewing TMAs using a microscope, for example, removal of background lighting variance and tint, and potential disorientation of the reviewer, which may have resulted in the discrepancies evident in the remaining three parameters

Population immunity to pneumococcal serotypes in Kilifi, Kenya, before and 6 years after the introduction of PCV10 with a catch-up campaign: an observational study of cross-sectional serosurveys

BACKGROUND: In Kilifi (Kenya), a pneumococcal conjugate vaccine (PCV10) was introduced in 2011 in infants (aged <1 year, 3 + 0 schedule) with a catch-up campaign in children aged 1-4 years. We aimed to measure the effect of PCV10 on population immunity. METHODS: In this observational study, repeated cross-sectional serosurveys were conducted in independent random samples of 500 children younger than 15 years every 2 years between 2009 and 2017. During these surveys, blood samples were collected by venesection. Concentrations of anti-capsular IgGs against vaccine serotypes (VTs) 1, 4, 5, 6B, 7F, 9V, 14, 18C, 19F, and 23F, and against serotypes 6A and 19A, were assayed by ELISA. We plotted the geometric mean concentrations (GMCs) by birth year to visualise age-specific antibody profiles. In infants, IgG concentrations of 0·35 μg/mL or higher were considered protective. FINDINGS: Of 3673 volunteers approached, 2152 submitted samples for analysis across the five surveys. Vaccine introduction resulted in an increase in the proportion of young children with protective IgG concentrations, compared with before vaccine introduction (from 0-33% of infants with VT-specific levels over the correlate of protection in 2009, to 60-94% of infants in 2011). However, among those vaccinated in infancy, GMCs of all ten VTs had waned rapidly by the age of 1, but rose again later in childhood. GMCs among children aged 10-14 years were consistently high over time (eg, the range of GMCs across survey rounds were between 0·45 μg/mL and 1·00 μg/mL for VT 23F and between 2·00 μg/mL and 3·11 μg/mL for VT 19F). INTERPRETATION: PCV10 in a 3 + 0 schedule elicited protective IgG levels during infancy, when disease risk is high. The high antibody levels in children aged 10-14 years might indicate continued exposure to vaccine serotypes due to residual carriage or to memory responses to cross-reactive antigens. Despite rapid waning of IgG after vaccination, disease incidence among young children in this setting remains low, suggesting that lower thresholds of antibody, or other markers of immunity (eg, memory B cells), may be needed to assess population protection among children who have aged past infancy. FUNDING: Gavi, the Vaccine Alliance; Wellcome Trust

Observations of ice multiplication in a weakly convective cell embedded in supercooled mid-level stratus

Simultaneous observations of cloud microphysical properties were obtained by in-situ aircraft measurements and ground based Radar/Lidar. Widespread mid-level stratus cloud was present below a temperature inversion (~5 °C magnitude) at 3.6 km altitude. Localised convection (peak updraft 1.5 m s−1) was observed 20 km west of the Radar station. This was associated with convergence at 2.5 km altitude. The convection was unable to penetrate the inversion capping the mid-level stratus. The mid-level stratus cloud was vertically thin (~400 m), horizontally extensive (covering 100 s of km) and persisted for more than 24 h. The cloud consisted of supercooled water droplets and small concentrations of large (~1 mm) stellar/plate like ice which slowly precipitated out. This ice was nucleated at temperatures greater than −12.2 °C and less than −10.0 °C, (cloud top and cloud base temperatures, respectively). No ice seeding from above the cloud layer was observed. This ice was formed by primary nucleation, either through the entrainment of efficient ice nuclei from above/below cloud, or by the slow stochastic activation of immersion freezing ice nuclei contained within the supercooled drops. Above cloud top significant concentrations of sub-micron aerosol were observed and consisted of a mixture of sulphate and carbonaceous material, a potential source of ice nuclei. Particle number concentrations (in the size range 0.1<D<3.0 μm) were measured above and below cloud in concentrations of ~25 cm−3. Ice crystal concentrations in the cloud were constant at around 0.2 L−1. It is estimated that entrainment of aerosol particles into cloud cannot replenish the loss of ice nuclei from the cloud layer via precipitation. Precipitation from the mid-level stratus evaporated before reaching the surface, whereas rates of up to 1 mm h−1 were observed below the convective feature. There is strong evidence for the Hallett-Mossop (HM) process of secondary ice particle production leading to the formation of the precipitation observed. This includes (1) Ice concentrations in the convective feature were more than an order of magnitude greater than the concentration of primary ice in the overlaying stratus, (2) Large concentrations of small pristine columns were observed at the ~−5 °C level together with liquid water droplets and a few rimed ice particles, (3) Columns were larger and increasingly rimed at colder temperatures. Calculated ice splinter production rates are consistent with observed concentrations if the condition that only droplets greater than 24 μm are capable of generating secondary ice splinters is relaxed. This case demonstrates the importance of understanding the formation of ice at slightly supercooled temperatures, as it can lead to secondary ice production and the formation of precipitation in clouds which may not otherwise be considered as significant precipitation sources

A photometric and spectroscopic study of the new dwarf spheroidal galaxy in Hercules

Our aim is to provide as clean and as complete a sample as possible of red giant branch stars that are members of the Hercules dSph galaxy. With this sample we explore the velocity dispersion and the metallicity of the system. Stromgren photometry and multi-fibre spectroscopy are combined to provide information about the evolutionary state of the stars (via the Stromgren c_1 index) and their radial velocities. Based on this information we have selected a clean sample of red giant branch stars, and show that foreground contamination by Milky Way dwarf stars can greatly distort the results. Our final sample consists of 28 red giant branch stars in the Hercules dSph galaxy. Based on these stars we find a mean photometric metallicity of -2.35 dex which is consistent with previous studies. We find evidence for an abundance spread. Using those stars for which we have determined radial velocities we find a systemic velocity of 45.2 km/s with a dispersion of 3.72 km/s, this is lower than values found in the literature. Furthermore we identify the horizontal branch and estimate the mean magnitude of the horizontal branch of the Hercules dSph galaxy to be V_0=21.17, which corresponds to a distance of 147 kpc. We have shown that a proper cleaning of the sample results in a smaller value for the velocity dispersion of the system. This has implications for galaxy properties derived from such velocity dispersions.Comment: 24 pages, 28 figure

International expert consensus on a scientific approach to training novice cardiac resynchronization therapy implanters using performance quality metrics

Aims: Pacing and Cardiac Resynchronization Therapy (CRT) procedural training for novice operators usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee ability to perform required tasks at predetermined performance levels prior to in-vivo practice. We sought to characterize and validate with experts a reference approach to pacing/CRT implants based on objective and explicit performance quality metrics, for the development of a reproducible, simulation-based, training curriculum aiming to operator proficiency. Methods: Three experienced CRT implanters, a behavioural scientist and two engineers performed a detailed task deconstruction of the pacing/CRT procedure and identified the performance metrics (phases, steps, errors, critical errors) that constitute an optimal CRT implant for training purposes. The metrics were stress tested to determine reliability and score-ability and then subjected to detailed systematic review by an international panel of 15 expert implanters in a modified Delphi process. Results: Thirteen procedure phases were identified, consisting of 196 steps, 122 errors, 50 critical errors. The expert panel deliberation added 16 metrics, deleted 12, and modified 43. Unanimous panel consensus on the resulting CRT procedure metrics was obtained, which verified face and content validity. Conclusion: A reference pacing/CRT procedure and metrics created by a core group of experts accurately characterize the essential components of performance and were endorsed by an international panel of experienced peers. The metrics will underpin quality-assured novice implanter training